Marek Petrik

I am an Associate Professor in the Department of Computer Science at the University of New Hampshire. I am also a member of the artificial intelligence research group at UNH.

Before UNH, I was a Research Staff Member at the IBM’s T. J. Watson Research Center in Yorktown, NY. I received my Ph.D. from University of Massachusetts Amherst in 2010. My advisor was: Shlomo Zilberstein. In 2022/2023 I was also a visiting faculty researcher at Google Research in Mountain View, CA.

Research Interests

I am interested in robust data-driven decision making and in particular in reinforcement learning with limited data sets, risk aversion, and Bayesian models of uncertainty. My application areas include natural resource management, agriculture, and space physics. I had previously worked on applications in precision agriculture, natural resources, renewable energy management, supply chains, advertising, and others.

Publications

Contact Information

If you would like to meet with me, please email me. However, please do not send me AI-generated emails. I generally ignore AI-generated emails.

Email: mpetrik@cs.unh.edu
Office Phone: +1-603-862-2682
Department of Computer Science, Kingsbury N215b, Durham, NH 03824

CS980: Advanced Machine Learning

Where

Kingsbury N233

When

MW 3:40pm - 5:00pm

What

In this seminar, we will cover how to make good decisions using machine learning. Reinforcement learning and multi-armed bandits are just some of the methods that combine decision making with machine learning. These methods play a crucial role in countless real problems such as when personalizing websites, navigating robots, managing supply chains and revenue, and even when playing Go!

We will cover how to trade off exploitation and exploration; that is when to act with the current data versus acquiring additional data to make a better decision later. The exploration/exploitation tradeoff is a key challenge in reinforcement learning. We will also cover the related issues of reliability and robustness in machine learning.

Some of the specific topics we will cover are:

A/B testing and randomized experiments
Optimizer’s curse
Thompson sampling, UCB, EXP3
(Robust) Markov decision processes
Gittins and Whittle indices
Tree and policy search
Covariate shift

The class will start with an overview of fundamental problems, methods, and techniques and will continue with student presentations and a research project.

Pre-requisites

Statistics and some linear algebra. Machine learning is not a pre-requisite. We will cover topics that are closely related to ML but do not depend on it.

CS980: Advanced ML: Reinforcement Learning

Please see the class homepage for schedule and other up-to-date information.

Where

Kingsbury N233

When

MW 3:40pm - 5:00pm

What

In this seminar, we will cover reinforcement learning, or how to make good decisions driven by data. The goal in reinforcement learning is to learn how to act while interacting with a dynamic and complex environment. Reinforcement learning methods can be applied in various domains, such as when managing ecosystems, optimizing website, to robotics, and healthcare.

Our focus will be on reinforcement learning that can learn from batch data sets without interacting with the environment. The algorithms have to learn how to interact with the environment based on a historical data. These batch methods are important when the cost of failure is high, such as in healthcare of agriculture, when using trial and error is impractical.

Some of the topics that we will cover are:

Markov decision processes: Policy iteration, Value iteration, Linear programming
Policy evaluation (online and offline): TD, LSTD
Batch policy improvement: Q-learning, LSPI
Convex optimization: Linear programming

The focus of the class will be on depth rather than breadth. The main goal of the class is delivering an interesting reinforcement learning group project. As much as possible we will work on the same code-base using a subset of C++, Python, and R. The class will require independent study of reading materials and in-class group problem solving and paper discussions.

Textbooks

Follow the links for free online versions.

Our main textbook will be:

Szepesvári, C. (2010). Algorithms for Reinforcement Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4(1), 1–103.

Additional material can be found in:

Puterman, M. L. (2005). Markov decision processes: Discrete stochastic dynamic programming. John Wiley & Sons, Inc.
Sutton, R. S., & Barto, A. (1998). Reinforcement learning. MIT Press.

Pre-requisites

Statistics, some linear algebra and ideally familiarity with machine learning.

More details

CS980: Advanced ML: Reinforcement Learning

Please see the class homepage for schedule and other up-to-date information.

Where and When

Kingsbury N233
MW 3:40pm - 5:00pm

What

This seminar will cover reinforcement learning. The goal in reinforcement learning is to learn how to act while interacting with a dynamic and complex environment. Instead of trying to cover all reinforcement learning topics, we will focus on the foundations needed to understand other concepts.

Markov decision processes: Policy iteration, Value iteration, Linear programming
Value Function Approximation: LSTD, LSPI, ALP, ABP
Uncertainty: Bandits and Robust Markov Decision Processes

We will use Python, R, or C++ and cover relevant topics from linear algebra, mathematical optimization, and statistics as needed.

More Info

Please see the class homepage.

CS980: Advanced ML: Reinforcement Learning

Please see the class homepage for more information.

Where and When

Kingsbury N233
TR 3:40pm - 5:00pm

What

Markov decision processes: Policy iteration, Value iteration, Linear programming

Standard RL methods: SARSA, TD, Q-learning
Model-based methods: Dyna and others
Value Function Approximation: LSTD, LSPI, ALP, ABP
Uncertainty: Exploration and Robustness

We will use Python, R, or C++ and cover relevant topics from linear algebra, mathematical optimization, and statistics as needed.

Textbook

Richard Sutton, Andrew Barto, Reinforcement Learning: An Introduction, 2018.

CS780 / CS880: Introduction to Machine Learning

When and Where

Tue & Thu, 12:40 pm - 2:00 pm Kingsbury N133

See class overview for more information on textbooks, syllabus, assignments, office hours, and grading.

Assignments

Please use Piazza for questions about assignments.

Assignment	Due Date
Assignment 1	2/14/17 at 12:40PM
Assignment 2	2/21/17 at 12:40PM
Assignment 3	3/09/17 at 12:40PM
Assignment 4	4/06/17 at 12:40PM
Assignment 5	4/20/17 at 12:40PM

Syllabus

Date	Slides	Reading	Notebooks
1/26	Statistical learning	ISL 1,2	(html) (RMD)
1/31	Linear regression I	ISL 3.1-2	(html) (RMD)
2/02	No class
2/07	Linear regression II	ISL 3.3-6
2/09	No class
2/14	Logistic regression	ISL 4.1-3	(html)(RMD)
2/16	LDA, QDA, Bayes	ISL 4.4-6
2/21	Cross-validation	ISL 5
2/23	Model selection	ISL 6.1-6.2
2/28	Dimensionality	ISL 6.3-6.4
3/2	PCA ML/MAP	ISL 10.1-2	ML PCA
3/6	Clustering and EM	ISL 10.3-5	kmeans
3/9	Midterm Review	ISL 1-6, 10
3/21	Midterm
3/23	Linear algebra	LAO 1.1-2,2,3
3/28	LA in ML	LAR	linear algebra
3/30	LA in ML	LAR	linear algebra
4/04	SVM	ISL 9
4/06	Decision trees and boosting	ISL 8
4/11	Nonlinear methods	ISL 7
4/13	Recommender systems
4/18	Bayes nets	MLP 10
4/20	Reinforcement learning	RL
4/25	Final exam review
4/27	Project presentations (Graduate)
5/02	Deep learning and big data	DL
5/04	Project presentations (Undergraduate)

Project

See the project overview for details on the details of deliverables. The deliverable are due by the end of the day (midnight).

Date	Deliverable	Page Limit
2/24	Project description and data sources	1
3/07	Evaluation methodology	1
3/23	Method and literature overview	2
4/06	Preliminary results	3
4/27	Final report	7

Exams

See practice questions for questions you should be able to answer to be ready for the midterm and final exams.

Date	Exam
3/21	Midterm (take home)

Textbooks

Main reference:

ISL: James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning

Class Content

The goal of this class is to teach you how to use machine learning to understand data and make predictions in practice. The class will cover the fundamental concepts and algorithms in machine learning and data science as well as a wide variety of practical algorithms. The main topics we will cover are:

The maximum likelihood principle

Regression: Linear regression
Classification: Logistic regression and linear discriminant analysis
Cross-validation, bootstrap, and over-fitting
Model selection: Regularization, Lasso
Nonlinear models: Decision trees, Support vector machines
Unsupervised: Principal component analysis, k-means
Advanced topics: Bayes nets and deep learning

The graduate version of the class will cover the same topics in greater depth.

Programming Language

The class will involve hand-on data analysis using machine learning methods. The recommended language for programming assignments is R which is an excellent tool for statistical analysis and machine learning. No prior knowledge of R is needed or expected; the book and lecture will cover a gentle introduction to the language. Experienced students may also choose other alternatives, such as Python or Matlab.

Pre-requisites

Basic programming skills (scripting languages like Python are OK) and some familiarity with statistics and calculus. If in doubt, please email me.

CS780 / CS880: Machine Learning

Please see the main class website for detailed up to date information.

When and Where

Mon & Wed, 11:10 am - 12:40 pm in Kingsbury N113

Class Content

The maximum likelihood principle

Regression: Linear regression
Classification: Logistic regression and linear discriminant analysis
Cross-validation, bootstrap, and over-fitting
Model selection: Regularization, Lasso
Nonlinear models: Decision trees, Support vector machines
Unsupervised: Principal component analysis, k-means
Advanced topics: Bayes nets and deep learning

The graduate version of the class will cover the same topics in greater depth.

Programming Language

We recommend using the free R Studio for completing programming assignments. R Notebooks are very convenient for producing reproducible reports and we encourage you to use them. Jupyter is a similar alternative for Python.